1. Field of the Invention
The present invention relates to an image retrieval system and an image retrieval method in which characteristic values and/or pieces of attribute information are extracted from pieces of image data such as moving images or static images recorded in analog or digital and in which the retrieval of desired image data is performed by using the extracted characteristic values and/or the pieces of extracted attribute information.
2. Description of Related Art
FIG. 1 shows the configuration of a system as an example of a conventional image retrieval processing system. This system is disclosed in a letter xe2x80x9cSynthesis and Evaluation of the Image Database with Fully Automated Keyword Extraction by State Transition Model and Scene Description Languagexe2x80x9d edited by Institute of Electronic Information and Communication Engineers of Japan, D-II Vol. J79-D-II No.4, pp.476-483, April of 1996. In this system, static images are processed in the image retrieval. That is, an area of each of images is divided into a plurality of segments in an area dividing unit 103 of a preparation unit 101, and a plurality of keywords are attached to each divided segment. As the keywords, a conception keyword and a scene description keyword are prepared. In a conception keyword extracting unit 104, a conception keyword 108 of each segment is obtained according to a color and a characteristic value of the segment by using conception keywords respectively allocated to color information in advance. In a scene description keyword describing unit 105, a predicate relating to xe2x80x9cpositionxe2x80x9d, xe2x80x9ccolorxe2x80x9d, xe2x80x9cshapexe2x80x9d, xe2x80x9csizexe2x80x9d, xe2x80x9cdirectionxe2x80x9d or the like is obtained from a plurality of image characteristic values of segments. In the unit 105, an operation 106, in which a user 107 selects one predicate from predicates defined in advance and describes the selected predicate, is required, and the selected predicate is output as a scene description keyword 109. In a retrieval tool 102, conception keywords and scene description keywords are prepared in advance. A user 111 selects one conception keyword and one scene description keyword as each of keywords 112 from the prepared keywords. In a characteristic identifying unit 110, an identity of characteristic values is checked by comparing each keyword 112 selected by the user 111 with the conception keyword 108 or the scene description keyword 109 attached to each segment of the image, and an image retrieval processing is performed for the image.
However, in the above-described image retrieval processing system, an identity of characteristic values is checked by using keywords such as conception keywords and scene description keywords selected by the user 112 and keywords attached to each image, and an image retrieval processing is performed according to the characteristic values of each image. Therefore, all images are searched according to only the characteristic values of the images, so that it takes a lot of time to retrieve a desired image.
Also, in the above-described image retrieval processing system, a description method or a storing method of each keyword is not considered. Therefore, it is required that a plurality of image servers relate to a plurality of retrieval tools denoting clients in one-to-one correspondence. As a result, a system, in which many users respectively perform the image retrieval through a network while using various retrieval tools, cannot be provided for the users.
Also, because only static images are processed in the image retrieval, it is difficult to retrieve a desired moving image.
The present invention is provided to solve the above problems, and a main object of the present invention is to provide an image retrieval system and an image retrieval method in which an image retrieval processing can be efficiently performed.
A subordinate object of the present invention is to provide an image retrieval system and an image retrieval method which does not depend on a plurality of image servers distributed in a network by describing and producing a plurality of retrieval keywords according to a common syntax.
Another subordinate object of the present invention is to provide an image retrieval system and an image retrieval method in which the retrieval of a desired moving image can be easily performed by extracting a characteristic value for each video segment, which is composed of a plurality of frames, in place of the extraction of a characteristic value for each frame when a plurality of keywords are extracted from moving images.
An image retrieval system according to the present invention, comprises a characteristic descriptor producing unit for extracting a plurality of image characteristic values from pieces of input image data and producing a characteristic descriptor for each piece of input image data, an image information storing unit for storing the characteristic descriptors produced in the characteristic descriptor producing unit while holding the correspondence of each characteristic descriptor to one piece of input image data, an attribute list producing unit for producing an attribute list according to a piece of attribute information attached to each piece of input image data, and an image retrieving unit for receiving a first retrieval condition relating to attribute information, searching the attribute list produced in the attribute list producing unit for one piece of attribute information conforming to the first retrieval condition, outputting the piece of attribute information conforming to the first retrieval condition, receiving a second retrieval condition relating to a characteristic descriptor, searching the image information storing unit for one piece of image data conforming to the second retrieval condition and outputting the piece of image data conforming to the second retrieval condition.
Therefore, the retrieval can be efficiently performed.
In an image retrieval system according to the present invention, the attribute list is produced according to a syntax, which defines a data structure of the attribute list, in the attribute list producing unit, and the piece of attribute information conforming to the first retrieval condition is retrieved according to the syntax of the attribute list in the image retrieving unit.
Therefore, the retrieval can be efficiently performed in a short time.
In an image retrieval system according to the present invention, the characteristic descriptors are produced according to a syntax, which defines a data structure of each characteristic descriptor, in the characteristic descriptor producing unit, and the piece of image data conforming to the second retrieval condition is retrieved in the image retrieving unit according to the syntax of the characteristic descriptors.
Therefore, the image retrieval not depending on a plurality of image servers distributed in the network can be performed.
In an image retrieval system according to the present invention, one image characteristic value is extracted in the characteristic descriptor producing unit for each frame, and one characteristic descriptor is produced in the characteristic descriptor producing unit for each video segment composed of a group of frames.
Therefore, the retrieval of a moving image can be easily performed.
In an image retrieval system according to the present invention, each piece of input picture data received in the characteristic descriptor producing unit denotes compressed video data which composes one or more intra-frames and one or more inter-frames, both an average value and a standard deviation are produced as one characteristic descriptor of the intra-frames of the video segment in the characteristic descriptor producing unit by extracting an average matrix of pixel values in a prescribed coding area of one intra-frame for each intra-frame of the video segment, calculating a sum of the average matrices of all intra-frames included in the video segment and calculating both the average value of the average matrices and the standard deviation of the average matrices from the sum and the number of intra-frames in the video segment, and one characteristic descriptor of the inter-frames of the video segment is produced in the characteristic descriptor producing unit by extracting a matrix of motion vectors of pixels of the prescribed coding areas of one inter-frame for each inter-frame of the video segment, calculating an average of the motion vectors of each inter-frame of the video segment as a motion vector average, classifying zero run lengths, which are obtained according to a threshold processing for the motion vectors of one inter-frame, into a plurality of classified types of zero run lengths for each inter-frame of the video segment, calculating an average of the motion vector averages and a plurality of classified types of average zero run lengths in the video segment according to the number of inter-frames in the video segment and setting the average of the motion vector averages and the classified types of average zero run lengths as the characteristic descriptor of the inter-frames of the video segment.
Therefore, the retrieval of a moving image can be easily performed.
In an image retrieval system according to the present invention, each piece of input picture data received in the characteristic descriptor producing unit denotes non-compressed video data, the characteristic values are extracted from the pieces of non-compressed video data in the characteristic descriptor producing unit to produce the characteristic descriptors, and the characteristic descriptors and pieces of compressed video data, which are obtained by compressing the pieces of non-compressed video data according to a prescribed video compression method, are stored in the image information storing unit while the correspondence of each characteristic descriptor to one piece of compressed video data is held.
Therefore, the retrieval can be efficiently performed.
In an image retrieval system according to the present invention, a prescribed condition equation stored in advance is read out by the image retrieving unit in cases where the image retrieving unit searches the image information storing unit for the piece of image data conforming to the characteristic descriptor specified by the second retrieval condition to output the piece of image data, and it is judged according to the prescribed condition equation whether or not the piece of image data conforms to the characteristic descriptor specified by the second retrieval condition.
Therefore, the retrieval can be efficiently performed.
In an image retrieval system according to the present invention, each piece of input image data denotes a monitored image recorded by a monitoring camera.
Therefore, a trespasser can be detected.
In an image retrieval system according to the present invention, each piece of input image data denotes an image of a stored video mail.
Therefore, a user can retrieve a desired mail.
In an image retrieval system according to the present invention, each piece of input image data denotes an image of a stored broadcast program.
Therefore, even though a user does not accurately remember program information, the user can retrieve a desired program according to an image impression.
In an image retrieval system according to the present invention, each piece of input image data denotes a video image recorded by a video camera.
Therefore, the image retrieval can be efficiently performed by using the similarity held in the images as a key.
An image retrieval method according to the present invention comprises the steps of extracting a plurality of image characteristic values from pieces of input image data to produce a characteristic descriptor for each piece of input image data, storing the produced characteristic descriptors while holding the correspondence of each characteristic descriptor to one piece of input image data, producing an attribute list according to a piece of attribute information attached to each piece of input image data, receiving a first retrieval condition relating to attribute information, searching the attribute list for one piece of attribute information conforming to the first retrieval condition, outputting the piece of attribute information conforming to the first retrieval condition, receiving a second retrieval condition relating to a characteristic descriptor, searching the pieces of stored image data for one piece of image data conforming to the second retrieval condition while referring to the piece of attribute information conforming to the first retrieval condition, and outputting the piece of image data conforming to the second retrieval condition.
Therefore, the image retrieval can be efficiently performed
An image retrieval system according to the present invention comprises a characteristic descriptor producing unit for extracting a plurality of image characteristic values from pieces of input image data and producing a characteristic descriptor for each piece of input image data, a for-retrieval description data producing unit for describing each characteristic descriptor as a piece of for-retrieval description data while holding the correspondence of the piece of for-retrieval description data to a space-time structure of the pieces of input image data, an image information storing unit for storing each piece of for-retrieval description data with the piece of input image data corresponding to the piece of for-retrieval description data, a first retrieval processing unit, a second retrieval processing unit, and a user interface unit. A retrieval request received from a user through the user interface unit is sent from the second retrieval processing unit to the first retrieval processing unit as retrieval request information, a retrieval result sent from the first retrieval processing unit is received in the second retrieval processing unit, the second retrieval processing unit presents the retrieval result to the user through the user interface unit, the pieces of for-retrieval description data of the pieces of input image data stored in the image information storing unit are analyzed in the first retrieval processing unit according to the retrieval request information sent from the second retrieval processing unit to extract the characteristic descriptors, a conformity judgment processing is performed in the first retrieval processing unit according to the extracted characteristic descriptors to obtain the retrieval result, and the retrieval result is sent from the first retrieval processing unit to the second retrieval processing unit to make the second retrieval processing unit present the retrieval result to the user through the user interface unit.
Therefore, the image data retrieval based on the character descriptors can be efficiently performed.
The image retrieval system according to the present invention further comprises a characteristic descriptor reliability calculating unit for calculating a degree of reliability of each characteristic descriptor produced in the characteristic descriptor producing unit. The for-retrieval description data producing unit describes each characteristic descriptor and the degree of reliability of the characteristic descriptor as one piece of for-retrieval description data while holding the correspondence of the piece of for-retrieval description data to the space-time structure of the pieces of input image data, the pieces of for-retrieval description data of the pieces of input image data stored in the image information storing unit are analyzed in the first retrieval processing unit according to the retrieval request information sent from the second retrieval processing unit to extract the characteristic descriptors and the degrees of reliability of the characteristic descriptors, a conformity judgment processing is performed in the first retrieval processing unit according to the extracted characteristic descriptors and the degrees of reliability of the characteristic descriptors to obtain the retrieval result, and the retrieval result is sent from the first retrieval processing unit to the second retrieval processing unit to make the second retrieval processing unit present the retrieval result to the user through the user interface unit.
Therefore, the retrieval can be efficiently performed.
In an image retrieval system according to the present invention, the necessity of the conformity judgment processing for each characteristic descriptor is estimated in the first retrieval processing unit according to the degree of reliability of the characteristic descriptor in case of the conformity judgment processing, and the conformity judgment processing for the characteristic descriptor is skipped in cases where it is judged that the conformity judgment processing for the characteristic descriptor is not necessary.
Therefore, the retrieval processing can be performed at high speed, and the retrieval efficiency can be moreover improved.
The image retrieval system according to the present invention further comprises a characteristic descriptor retrieval order determining unit for determining a use order of the characteristic descriptors in the retrieval according to the degrees of reliability calculated in the characteristic descriptor reliability calculating unit. The for-retrieval description data producing unit describes each characteristic descriptor, the degree of reliability of the characteristic descriptor and the use order in the retrieval as one piece of for-retrieval description data while holding the correspondence of the piece of for-retrieval description data to the space-time structure of the pieces of input image data, the pieces of for-retrieval description data of the pieces of input image data stored in the image information storing unit are analyzed in the first retrieval processing unit according to the retrieval request information sent from the second retrieval processing unit to extract the characteristic descriptors, the degrees of reliability of the characteristic descriptors and the use order in the retrieval, a conformity judgment processing is performed in the first retrieval processing unit according to the extracted characteristic descriptors, the degrees of reliability and the use order in the retrieval to obtain the retrieval result, and the retrieval result is sent from the first retrieval processing unit to the second retrieval processing unit to make the second retrieval processing unit present the retrieval result to the user through the user interface unit.
Therefore, the image data retrieval based on the characteristic descriptors, the degrees of reliability and the use order in the retrieval can be efficiently performed.
In an image retrieval system according to the present invention, the retrieval request information, in which a use order of the characteristic descriptors in the retrieval, is sent from the second retrieval processing unit to the first retrieval processing unit as the retrieval request sent from the user through the user interface unit, and a retrieval order of the characteristic descriptors is renewed in the first retrieval processing unit according to the use order in the retrieval which is included in the retrieval request information sent from the second retrieval processing unit to perform a retrieval processing corresponding to the retrieval request of the user.
Therefore, the retrieval processing corresponding to the retrieval request of the user can be performed.
In an image retrieval system according to the present invention, the number of characteristic descriptor candidates in the retrieval result obtained in the conformity judgment processing is determined according to the degrees of reliability of the characteristic descriptors in the first retrieval processing unit for each of a plurality of retrieval steps of which an applied order is determined according to the use order in the retrieval, and the conformity judgment processing is performed.
Therefore, the retrieval processing can be performed at high speed.
The image retrieval system according to the present invention further comprises a video data reproducing server, and a video data decoding and reproducing unit. Each piece of input image data denotes a piece of video data, a piece of key image data representing each of pieces of video data specified in the retrieval is sent from the first retrieval processing unit to the second retrieval processing unit as the retrieval result, each piece of key image data is received in the second retrieval processing unit as the retrieval result sent from the first retrieval processing unit to present the retrieval result to the user through the user interface unit, a reproduction request indicating a specific key image, which is selected by the user from a plurality of key images indicated by the pieces of key image data, is sent from the user interface unit and is received in the video data reproducing server, a piece of specific video data represented by the specific key image is read out from the image information storing unit and is sent to the video data decoding and reproducing unit, and the piece of specific video data sent from the video data reproducing server is received in the video data decoding and reproducing unit, is decoded and is received.
Therefore, a total amount of transferred information can be effectively reduced.
An image retrieval method according to the present invention comprises the steps of extracting a plurality of image characteristic values from pieces of input image data to produce a characteristic descriptor for each piece of input image data, describing each characteristic descriptor as a piece of for-retrieval description data while holding the correspondence of the piece of for-retrieval description data to a space-time structure of the pieces of input image data, storing each piece of for-retrieval description data with the piece of input image data corresponding to the piece of for-retrieval description data, analyzing the pieces of stored for-retrieval description data of the pieces of input image data according to a retrieval request sent from a user to extract the characteristic descriptors, performing a conformity judgment processing according to the extracted characteristic descriptors to obtain a retrieval result, and presenting the retrieval result to the user.
Therefore, the image data retrieval based on the characteristic descriptors, the degrees of reliability and the use order in the retrieval can be efficiently performed.
The image retrieval method according to the present invention further comprises the steps of preparing pieces of video data as the pieces of input image data, presenting a piece of key image data, which represents each of pieces of video data specified in the retrieval, to the user as the retrieval result, receiving a reproduction request indicating a specific key image which is selected by the user from a plurality of key images indicated by the pieces of key image data, reading out a piece of specific video data represented by the specific key image, and decoding and reproducing the piece of specific video data.
Therefore, a total amount of transferred information can be effectively reduced.